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FOREWORD 


This  technical  report  describes  work  conducted  as  part  of  the  Navy  Personnel  Research  and 
Development  Center’s  Communication  Networks  in  Training  (CNIT)  project  in  the  general  area  of 
remote-site  training.  The  CNIT  project  is  one  part  of  the  Schoolhouse  Training  product  line  and 
falls  under  the  Personnel  and  Training  Technology  (NP2A)  Block  of  the  6.2  Mission  Support 
Technology  Program  Element  0602233N  (Work  Unit  No.  RM33T23.02).  The  work  was  performed 
under  the  sponsorship  of  the  Office  of  Naval  Technology. 

The  objective  of  the  project  is  to  find  more  cost-effective  ways  to  train  personnel  who  are 
geographically  remote  from  training  resources.  The  project  has  been  exploring  the  use  of  new 
communication  technologies  to  export  training  to  geographically-remote  students.  Among 
these  technologies  are  computer  networking,  two-way  video,  facsimile,  and  other  media. 

This  technical  report  describes  work  carried  out  during  the  initial  phases  of  the  project.  It  is 
intended  to  document  work  performed  and  provide  a  sketch  of  the  computer-based  prototype 
instructional  support  network  evaluated  during  FY89. 

The  authors  gratefully  acknowledge  the  help  of  the  Naval  Postgraduate  School  (NPS)  in  the 
conduct  of  the  research  reported  in  this  document.  We  are  particularly  indebted  to  Dr.  Robert 
Zucker,  director  of  Continuing  Education  through  spring  of  1989;  Mr.  Theodore  Calhoon,  Dr. 
Zucker’s  successor;  Drs.  Maurice  Weir  and  William  Little  of  the  Mathematics  Department;  Mr. 
Douglas  Williams  and  Ms.  Carolyn  Miller  of  the  NPS  computer  center;  and  the  continuing 
education  students  who  participated  in  the  study. 

Some  software  used  in  the  Instructional  Support  Network  was  developed  under  contract 
N6601-88-D-0054  by  Systems  Engineering  Associates  (SEA),  Inc. 

The  recommendations  in  this  technical  report  are  intended  for  use  by  the  Chief  of  Naval 
Operations  (OP-11)  and  the  Chief  of  Naval  Education  and  Training  in  developing  policy  for  the 
application  of  advanced  communication  technology  in  Navy  training. 


B.  E.  BACON  R.  C.  SORENSON 

Captain,  U.S.  Navy  Technical  Director  (Acting) 

Commanding  Officer 
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SUMMARY 


Background  and  Problem 

A  requirement  exists  to  train  Navy  personnel  who  are  geographically  remote  from  training 
resources.  This  requirement  exists  due  to  the  dispersal  of  home  ports,  training  facilities,  fleet  units, 
and  Naval  Reserve  detachments.  Evolving  technologies  have  the  potential  to  reduce  the  impact  of 
geography  on  training.  For  example,  computer  networking  can  link  together  student  and  instructor 
across  great  distances  and  across  time.  The  solution  to  the  Navy’s  remote-site  training  problem  lies 
in  the  proper  selection  and  use  of  new  communication  technologies. 

Objective 

The  overall  objective  of  the  project  is  to  explore  more  cost-effective  wave  to  train  personnel 
who  are  geographically  remote  from  training  resources.  The  objective  of  the  work  described  in  this 
technical  report  was  to  develop,  test,  and  evaluate  an  experimental,  computer-based  Instructional 
Support  Network  (ISN).  The  ISN  consists  of  an  electronic  communication  network  linking 
students  to  instructional  resources  such  as  instructors  and  instructional  materials. 

Approach 

A  preliminary  ISN  concept  was  developed  and  then  refined  and  focused  in  a  series  of  steps. 
These  steps  included  a  survey  of  related  projects  and  research  and  development  efforts,  selection 
of  a  research  testbed,  design  and  development  of  a  prototype  ISN,  and  conduct  of  a  field  test. 

Findings 

Formative  Evaluation 

Subjects  were  successfully  able  to  unpack  their  ISN  system,  set  it  up,  get  it  running,  receive  a 
message,  print  out  a  message,  compose  a  message,  and  send  a  message  within  less  than  two  hours. 

Process  Measures 

Student  Message  Traffic.  The  majority  of  ISN  messages  (72.9%)  were  administrative,  17.3% 
were  technical,  and  9.8%  were  motivational.  Students  initiated  40.9%  of  messages,  and  the  tutor 
initiated  the  remainder. 

ISN  Support  Requirements.  ISN  support  personnel  included  a  full-time  tutor,  part-time 
software  engineer,  occasional  “hot-line”  support,  and  occasional  support  from  the  NPS  host 
computer.  The  technical  support  by  the  software  engineer  averaged  one  to  two  hours  per  week;  hot¬ 
line  support,  about  30  minutes  per  week;  and  computer  center  support,  about  30  minutes  per  week. 

The  tutor  estimated  that  he  could  have  tutored  as  many  as  40  students  with  the  ISN,  and  perhaps 
more. 

Tbtor  Comments.  The  tutor  characterized  the  ISN  as  a  “great”  tutoring  tool  that  both  students 
and  tutor  found  easy  to  learn  to  use.  ISN  problems  identified  by  the  tutor  included  slow  student 


progress  with  lessons,  lack  of  communication  by  students,  NPS  host  computer  problems  that 
precluded  student-tutor  communication,  DDN  problems  that  precluded  student-tutor 
communication,  and  inability  (by  tutor)  to  diagnose  cause  of  student  noncommunication. 
Suggestions  for  improving  the  ISN  included  providing  students  with  a  syllabus  containing  a 
suggested  schedule  for  progress  through  their  course,  working  with  students’  work 
supervisors  to  allow  them  to  “ease  off”  on  their  duties  to  provide  more  time  for  their  studies,  and 
providing  additional  capabilities  within  the  ISN  software. 

Technical  Problems.  The  ISN  is  a  complex  arrangement  of  computers,  software,  and  long¬ 
distance  networks  whose  operability  depends  on  all  components  working  properly;  the  failure  of 
any  single  component  usually  compromises  the  entire  system.  While  the  ISN  was  in  use,  nearly 
every  component  in  it  failed  at  least  once.  Most  failures  were  minor  and  were  quickly  corrected, 
but  a  few  affected  the  ISN  for  several  weeks  and  required  software  modifications.  ISN  software 
and  hardware  were  reliable  and  robust;  the  most  serious  problems  were  the  result  of  system 
elements  beyond  the  immediate  control  of  ISN  support  personnel,  e.g.,  NPS  host  hardware  or 
software  downtime,  or  MILNET  downtime. 

Formal  Evaluation 

Some  evidence  shows  that  use  of  the  ISN  reduced  student  attrition  and  was  particularly  helpful 
to  students  with  weak  mathematics  backgrounds;  this  conclusion  was  not  supported  by  tests  of 
statistical  significance,  however. 

Students  who  used  the  ISN  rated  their  tutor’s  knowledge  of  mathematics  and  tutoring  skill 
more  highly  than  students  undergoing  conventional  instruction. 

Conclusions 

The  use  of  electronic  mail  appears  to  have  limited  application  for  instructional  delivery  in  the 
Navy.  However,  an  electronic  mail  system  with  the  simplicity  and  user  friendliness  of  the  ISN 
would  be  a  major  improvement  over  existing  electronic  mail  systems  and  would  be  useful  for 
sharing  information  among  training  professionals  and  other  Navy  personnel. 

Recommendations 

1.  The  use  of  electronic  mail  should  not  be  considered  as  a  primary  instructional  delivery 
medium  by  the  Navy. 

2  The  Chief  of  Naval  Education  and  Training  should  give  serious  consideration  to 
developing  a  Navy-wide,  DDN-based,  electronic-mail  network  for  sharing  information 
(“knowledge  networking”)  among  training  professionals  and  other  Navy  personnel.  The 
design  of  this  network  could  be  modeled  on  the  ISN. 
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INTRODUCTION 


Problem  and  Background 

A  requirement  exists  to  train  Navy  personnel  who  are  geographically  remote  from  training  re¬ 
sources.  This  requirement  exists  throughout  the  Navy,  but  is  perhaps  most  obvious  for  personnel 
aboard  ships  at  sea.  Shipboard  training  is  limited  by  available  training  resources  and  the  skills  of 
shipboard  trainers.  By  necessity,  personnel  are  periodically  assigned  to  formal  schools  to  receive 
training  they  cannot  receive  aboard  ship.  Strategic  homeporting  may  disperse  Navy  ships  over  a 
greater  number  of  smaller,  more  geographically  isolated  ports.  As  training  resources  and  personnel 
tend  to  concentrate  near  larger  ports,  strategic  homeporting  can  be  expected  to  accentuate  the  train¬ 
ing  problem.  Even  without  the  homeporting  issue,  the  locations  of  existing  training  facilities  often 
require  fleet  personnel  to  travel  away  from  their  home  duty  station  to  complete  required  training. 
The  remote-site  training  requirement  also  exists  in  the  Naval  Reserves.  Reservists  typically  belong 
to  small  detachments,  widely  dispersed  geographically,  with  limited  training  resources,  few  quali¬ 
fied  trainers,  and  little  time  to  train.  The  requirement  to  overcome  geographic  distance  in  training 
delivery  is  a  generic  problem  that  exists  in  the  civilian  public  education  and  industrial  world  as  well 
as  in  the  military  world. 

Evolving  technologies  have  the  potential  to  reduce  the  impact  of  geography  on  training.  For 
example,  computer  networking  can  link  students  and  instructors  across  great  distances.  Asynchro¬ 
nous  communication  with  electronic  mail  permits  communication  across  time.  Programs  with  ar¬ 
tificial  intelligence  can  be  used  to  support  training,  reducing  the  need  for  live  instructors  and  the 
load  on  existing  instructors.  These  technologies  are  becoming  increasingly  available  and  have  the 
potential  to  reduce  the  Navy’s  remote  site  training  problems. 

The  solution  to  the  Navy’s  remote-site  training  problem  lies  in  the  proper  selection  and  use  of 
new  communication  technologies.  In  general,  these  technologies  are  costly  and  constantly  chang¬ 
ing;  new  technologies  appear  regularly.  Proponents  of  one  technology  or  another  proclaim  the  vir¬ 
tues  of  their  favorite.  Many  technologies  are  being  used  on  a  regular  basis;  others,  in  demonstration 
projects.  Investigators  are  exploring  strengths  and  limitations,  cost  effectiveness,  and  other  dimen¬ 
sions  governing  suitability  for  different  applications.  Unfortunately,  there  is  no  road  map  to  follow 
to  determine  which  technology  is  “best”  in  a  particular  application  today  and  which  will  be  “best” 
next  week,  next  year,  or  ten  years  from  now.  The  Communication  Networks  in  Training  (CNIT) 
project  is  exploring  different  technologies,  research  and  development  projects,  and  the  Navy’s 
training  problems  in  order  to  gain  a  better  understanding  of  which  technologies  hold  the  greatest 
potential  for  future  use  in  the  Navy. 

Objective 

The  primary  objective  of  the  CNIT  project  is  to  find  more  cost-effective  technological  ap¬ 
proaches  to  train  personnel  who  are  geographically  remote  from  training  resources.  This  objective 
is  being  addressed  along  four  different  tracks: 

1.  Assess  the  applicability  of  new  communication  technologies  to  the  solution  of  Navy 
training  problems  caused  by  geographic  dispersal  of  training. 
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2.  Develop,  test,  and  evaluate  an  experimental  computer-based  instructional  support  network. 

3.  Develop,  test,  and  evaluate  an  experimental,  two-way  videoteletraining  system 

4.  Investigate  the  impact  of  alternative  equipment  configurations  and  training  delivery 
methods  on  training  effectiveness  in  a  videoteletraining  laboratory. 

This  technical  report  describes  the  work  performed  on  track  2  during  FY88  and  FY89.  The 
track  2  work  concerned  the  design,  development,  and  evaluation  of  a  simple,  computer-based  in¬ 
structional  support  network  intended  for  use  by  personnel  with  little  or  no  computer  experience. 
Other  project  documentation  describes  work  performed  during  the  same  periods  on  tracks  1  and  3 
(see  Simpson,  1990;  Simpson,  Pugh,  &  Parchman,  in  press).  The  work  performed  on  track  4  is 
commencing  during  FY90  and  will  be  reported  in  the  future. 

APPROACH 


Overview 

The  project  began  with  the  development  of  a  preliminary  Instructional  Support  Network  (ISN) 
concept.  This  concept  was  then  evaluated,  refined,  and  focused  in  a  series  of  steps.  These  steps  in¬ 
cluded  a  survey  of  related  projects  and  research  and  development  efforts,  selection  of  a  research 
testbed,  design  and  development  of  a  prototype  ISN,  and  conduct  of  a  field  test. 

Preliminary  ISN  Concept 

The  preliminary  ISN  concept  was  a  very  simple,  relatively  inexpensive,  user-friendly  system 
to  link  students  to  an  instructor  across  geographic  distance  and  time.  The  concept  was  based  on 
electronic  mail,  but  was  to  incorporate  features  not  normally  found  in  mail  systems  and  to  be  much 
easier  to  use  than  traditional  mail  systems. 

The  preliminary  ISN  concept  consisted  of  three  elements: 

1.  A  communication  network  linking  the  student  to  instructional  resources  (e.g.,  instructor, 
training  materials,  question  and  answer  database). 

2.  Simple,  easy  to  use  workstations  for  student  and  instructor. 

3.  Operating  procedures,  documentation  and  technical/logistical  support. 

The  first  element  of  the  ISN  concept  was  predicated  on  the  use  of  computers  and  computer  net¬ 
working.  (The  ISN  concept  can  be  extended  to  use  video  or  other  media,  but  that  was  not  the  in¬ 
tention  of  this  project.)  Thr  ISN  would  be  built  to  work  within  the  Defense  Data  Network  (DDN), 
which  is  already  in  place  and  widely  used  for  data  transfer  and  electronic  mail.  The  DDN  is  de¬ 
scribed  in  greater  detail  in  the  appendix. 

The  second  element  of  the  ISN  concept,  while  straightforward,  is  unusual  in  terms  of  traditional 
electronic  mail  systems.  Most  of  these  systems  are  intended  for  use  by  personnel  with  considerable 
computer  sophistication  and  who  have  ready  access  to  technical  support.  The  ISN  concept  called 
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for  a  much  simpler  system  that  would  be  easy  to  master  and  use,  and  requires  minimal  external 
support. 

The  third  element  of  the  ISN  concept  is  the  procedures,  documentation,  and  support  system 
needed  to  make  ISN  work. 

The  preliminary  ISN  concept  did  not  define  the  instructional  setting,  although,  again,  it  was  im¬ 
plicit  that,  owing  to  its  networking  features  and  ability  to  bridge  distance,  the  ISN  would  have  its 
greatest  payoff  for  remote- site  instructional  applications,  where  student  and  instructional  resources 
were  far  apart.  There  was  nothing  to  preclude  its  use  in  the  confines  of  a  classroom,  but  this  did  not 
appear  to  capitalize  on  its  greatest  asset.  Because  of  the  intended  simplicity  of  the  ISN,  it  would 
also  have  applications  outside  of  instruction.  It  could  be  used,  for  example,  to  enable  knowledge 
networking  among  individuals  who  were  physically  separated  but  who  wished  to  share  informa¬ 
tion.  Unlike  most  electronic  mail  systems,  the  ISN  would  be  relatively  easy  to  learn  and  use. 

Survey  Related  Projects 

A  literature  review  was  conducted  using  the  Manpower  and  Training  Research  Information 
System  (MATRIS)  and  computer  data  bases  for  ERIC,  the  National  Technical  Information  System 
(NTIS),  and  Psychlnfo.  Key  topics  and  subtopics  were  computer  networking  (networks  in  training/ 
instruction,  computer  conferencing,  new  networking  technologies),  and  distance  education/train¬ 
ing.  Current  information  was  obtained  on  some  subtopics  from  articles  published  electronically 
and  in  magazines. 

To  avoid  duplication  with  other  projects,  we  made  a  strong  effort  to  identify  projects  using 
technology  similar  to  the  ISN.  Contact  was  made  with  researchers  in  the  Navy,  Army,  and  Air 
Force  training  communities  to  discuss  research  agendas  and  projects.  Through  this  process,  several 
projects  were  identified  and  investigated. 

Computer  networking  is  defined,  for  purposes  of  this  discussion,  as  the  linking  together  of  sep¬ 
arate  computer  nodes  (e.g.,  terminals,  workstations)  such  that  text  files  may  be  transferred  among 
them  under  user  control,  stored  locally,  and  accessed  at  will.  Most  networks  do  not  transfer 
video,  sound,  or  communication  modalities  other  than  text  but  this  constraint  is  eroding  rapidly 
with  the  advent  of  plug-in  cards  that  enable  computers  to  transfer  facsimiles  and  still-  or  slow- 
frame  video  using  fairly  narrow  bandwidth  telephone  lines,  e.g.,  64  kbps  (kilobits  per  second). 

Networks  enable  their  users  to  communicate  across  both  geographic  distance  and  time.  Users 
do  not  have  to  be  at  the  same  location  or  working  in  the  same  time  frame;  a  message  dropped  into 
an  electronic  mailbox  will  be  read  where  received  when  its  recipient  reads  the  computer  mail.  Net¬ 
working  links  computers  via  local,  regional,  national,  or  worldwide  networks,  which  give  their  us¬ 
ers  access  to  he  human  and  computer-based  resources  of  the  network. 

Computer  networks  are  a  relatively  recent  phenomenon.  They  began  in  the  early  1970s  but 
have  grown  steadily  so  that  presently  they  are  accessible  on  most  university  campuses,  government 
laboratories,  and  private  firms  involved  in  high  technology  and/or  government  contracting. 
Their  primary  use  is  for  communication  and  information  sharing  and  their  primary  users  are  tech¬ 
nical  professionals  and  managers. 
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.Among  the  most  prominent  researchers  of  user  interactions  with  networks  are  Hiltz,  Turoff, 
and  Kerr.  Hiltz  began  her  work  in  the  ear'y  1 970s  with  several  studies  of  the  Electronic  Information 
Exchange  System  (EIES),  alone  and  in  collaboration  with  the  other  two  authors  (Hiltz  &  Turoff, 
1978;  Turoff  &  Hiltz,  1978;  Kerr  &  Hiltz,  1982).  This  early  work  is  today  recognized  as  seminal 
in  the  field.  Hiltz  and  colleagues  found  several  factors  to  be  predictive  of  success  in  the  introduction 
of  a  network  to  a  new  user  community  (Hiltz,  Kerr,  &  Johnson,  1985).  The  factors,  in  order  of  pri¬ 
ority,  are: 

1.  Users  must  want  system  at  outset. 

2.  A  “critical  mass”  of  users  must  be  interested  in  communicating  with  their  peers. 

3.  Strong,  active  leadership  (e.g,  management  support)  must  be  present. 

4.  The  system  must  have  adequate  features,  a  good  user  interface,  and  high  reliability. 

5.  Users  must  be  adequately  trained. 

6.  Technical  support  must  be  readily  available. 

Hiltz  et  al.  found  that  perceived  pressure  to  use  the  system  was  somewhat  negatively  correlated 
to  use  of  the  network.  Perhaps  the  most  surprising  finding  was  that  typing  ability  was  not  predictive 
of  system  use. 

Most  research  into  the  use  of  networks  has  involved  sophisticated  users;  i.e.,  mature,  well-ed¬ 
ucated  individuals  with  high  verbal  and  reading  skills  who  were  self-motivated  to  use  the  system 
and  who  initiated  communications  without  outside  coercion.  This  is,  in  essence,  the  community  of 
the  scientific  researcher  and  is  a  somewhat  unique  social  group.  Because  of  these  unique  properties, 
questions  have  been  raised  about  the  applicability  of  the  technology  to  other  social  groupings  in, 
for  example,  the  classroom.  One  of  the  central  questions  is  how  will  other  user  groups  (e.g.,  young¬ 
er,  less  well  educated,  limited  reading/verbal  skills,  external  locus  of  control)  interact  with  net¬ 
works?  Another  question  is  how  can  a  network  be  used  to  deliver  training?  The  second  question 
raises  several  other  questions.  For  example,  how  does  the  interaction  protocol  affect  learning  vari¬ 
ables?  How  should  the  system  be  designed  to  make  it  most  effective  (e.g.,  type  of  workstation,  user 
interface,  modes  of  interaction,  etc.)?  Research  is  just  now  beginning  to  explore  some  of  these 
questions  systematically. 

To  date,  networks  have  seen  limited  use  in  education  and  training,  although  their  use  is  grow¬ 
ing.  Relatively  few  studies  deal  with  the  application  of  networking  technology  to  instruction. 
Those  that  do  are  generally  informal  reports  of  case  studies  or  loosely  controlled  studies  whose 
generalizability  is  limited.  For  example,  Barnes,  Swehosky,  and  Laguna-Castillo  (1988)  report  on 
a  classroom  experiment  involving  the  use  of  a  local  area  network  (LAN)  to  support  a  statistics 
course;  students  expressed  a  liking  for  the  new  method  of  instruction,  but  performance  was  on  a 
par  with  classroom  instruction.  Levin  and  colleagues  have  experimented  with  the  use  of  networks 
in  the  elementary  school  classroom.  In  one  study,  students  in  classrooms  in  the  U.S.,  Mexico,  Ja¬ 
pan,  and  Israel  were  linked  by  network  and  participated  in  group  problem-solving  tasks  (Levin, 
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1985;  Levin,  Riel,  Miyake,  &  Cohen,  1986).  The  authors  reported  that  the  instructional  environ¬ 
ment  helped  students  gain  insight  into  problems  more  fully  than  they  did  in  a  conventional  class¬ 
room. 

In  a  related  study,  analysis  of  electronic  mail  traffic  in  a  networked  instructional  environment 
found  that  instructor-student  discourse  had  more  “strands”  and  persisted  longer  than  in  the  class¬ 
room  (Quinn,  Mehan,  Levin,  &  Black,  1983). 

An  interesting  recent  research  project  in  the  military  setting  is  the  Army  Research  Institute 
(ARI)  Asynchronous  Computer  Conferencing  project  (Richards  &  Phelps,  1987).  Work  began  in 
1986  and  is  ongoing.  Computer  conferencing  over  the  Army  Forum  Network  is  used  to  link 
geographically  remote  students  to  an  instructor  and  to  each  other  for  discussion  in  an  electronic 
classroom  and  with  smaller  working  groups.  During  the  first  two  years,  subject  matter  was  the  U.S. 
Army  Engineer  Officer  Advanced  course,  the  instructor  was  an  honors  graduate  of  the  course  and 
an  Army  Reserve  Officer  who  was  hired  under  contract,  and  students  were  Army  Reserve  Officers 
(Hagman,  1988).  Training  materials  were  developed  from  those  used  in  the  corresponding  resident 
course  (Hahn,  1988).  Primary  instructional  delivery  was  with  correspondence-type  training  mate¬ 
rials,  supported  by  computer  conferencing  for  discussion.  Students  were  paced  through  materials 
at  a  group  rate.  Students  were  also  provided  with  text-editing  software  for  use  in  preparing  written 
assignments. 

Hahn  (1988)  reported  that  students  quickly  became  bored  when  a  single  medium  of  presenta¬ 
tion  was  used  and  opted  for  variety  in  presentations  to  sustain  interest.  Project  participants  have 
reported  their  experiences  and  several  “lessons  learned”  (Richards  &  Phelps,  1987;  Phelps  &  Ri¬ 
chards,  1987;  Richards,  1988).  Among  the  problem  areas  were  hardware  breakdowns,  student  dif¬ 
ficulties  in  setting  up  PC-XT  computer  equipment,  lesson  pacing  problems,  and  the  need  for 
training  both  instructors  and  students  in  the  rules  of  computer  conferencing.  The  ARI  project  has 
solved  these  and  many  other  practical  problems  of  the  electronic  classroom  and  offers  useful  les¬ 
sons  to  educators  and  trainers  working  in  this  area.  Two  additional  significant  contributions  are  the 
design  of  a  metaphorical  user  interface  for  students’  computers  (Halbert,  1988)  and  considerable 
well-documented  experience  in  the  training  t  'instructors  (Kaplan  &  Jones,  1988). 

Educational  networks  are  widely  used  in  colleges  and  universities.  We  did  not  conduct  a 
survey  but  are  aware  of  several  cases  which  are  probably  typical.  The  U.S.  Air  Force  Academy  uses 
a  network  to  tie  together  students  and  faculty.  All  students  have  access  to  terminals  (actually  Zenith 
248  computers1 ).  The  system  provides  e-mail  and  there  are  plans  to  use  it  for  delivery  of  computer- 
aided  instructional  programs  (Regian,  1988).  The  U.S.  Naval  Postgraduate  school  employs  a  sim¬ 
ilar  LAN.  Use  of  the  network  depends  upon  the  academic  major  and  the  particular  instructor;  there 
is  no  standard  way  to  use  the  network  in  instruction  (Buoni,  1988).  Some  instructors  do  not  use  it. 
Others-particularly  in  more  technical  majors-use  it  to  transmit  homework  assignments  and  solu¬ 
tions  to  exercises,  and  to  communicate  with  individual  students.  These  uses  are  informal  in  the 
sense  that  they  are  natural  outgrowths  of  the  e-mail  system  and  supplant  other  means  of  commu¬ 
nication  such  as  dropping  notes  in  mailboxes,  providing  handouts  in  class,  meeting  during  offices 
hours.  Informal  uses  of  a  LAN  do  not  include  delivery  of  instruction  or  computer  conferencing. 
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Identification  of  specific  equipment  and  software  is  for  documentation  only  and  does  not  imply  endorsement. 
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Many  colleges  arid  universities-particularly  those  directed  at  students  who  hold  full-time 
jobs— now  permit  students  to  take  some  classes  by  computer;  that  is,  the  class  is  essentially  a  com¬ 
puter  conference.  The  Western  Behavioral  Sciences  Institute  (WBSI)  in  La  Jolla,  CA,  has  for  sev¬ 
eral  years  used  computer  conferencing  to  link  together  high-level  managers  at  remote  sites  to  study 
and  discuss  strategic  planning  in  a  course  combining  classroom  meetings  and  electronic  meetings. 
These  educational  applications  are  more  formal  than  those  just  discussed  and,  in  fact,  involve  wide- 
area  networks  rather  than  LANs.  However,  such  applications  are  w  :nin  the  capability  of  any  insti¬ 
tution  possessing  a  LAN  even  if  they  are  not  present  policy.  While  the  use  of  LANs  appears  to  be 
growing  in  institutions  of  higher  learning,  we  have  found  no  evidence  that  this  is  happening  in  pri¬ 
mary  or  secondary  schools,  technical/trade  schools,  or  within  the  military  schoolhouse  training  en¬ 
vironment. 

Within  the  Navy  educational/training  community,  computer  networking-more  specifically, 
electronic  mail— has  grown  steadily  but  not  without  problems.  Today,  the  Navy  research  and  de¬ 
velopment  centers  can  communicate  using  the  DDN,  but  most  of  the  Navy  schools  are  not  linked 
into  the  system.  The  Navy  education  and  training  community  does  not  use  electronic  mail  in  sup¬ 
port  of  instruction.  Although  institutions  such  as  the  Naval  Postgraduate  School  use  electronic  mail 
to  some  extent,  its  use  is  idiosyncratic  at  best.  Traditionally,  the  primary  users  of  computer  net¬ 
works  and  electronic  mail  have  been  scientists  and  educators.  Electronic  mail  is  beginning  to  find 
its  way  into  schools,  particularly  at  the  college  and  graduate  level.  The  common  denominators  of 
these  applications  are  mature,  motivated  use-  '  ''ho  take  the  initiative  in  using  the  medium. 

Beyond  its  possible  application  as  an  instructional  support  medium,  electronic  mail  can  also  be 
used,  as  noted  earlier,  for  knowledge  networking  among  professional  groups  (e.g.,  educational  spe¬ 
cialists,  aviators,  supply  officers).  Currently,  the  Navy  has  no  formal  policy  or  plan  for  using 
electronic  mail  in  this  way.  This  is  in  contrast  to  the  Army,  which  established  the  Army  Forum  Net¬ 
work  (AFN)  in  1983  to  “link  via  computer  teleconferencing  geographically  dispersed,  multidisci¬ 
plinary  teams  capable  of  rapidly  integrating  the  flow  of  critical  information  to  enhance  the  total 
Army  Mission”  (Department  of  the  Army,  1986).  As  of  late  1986,  AFN  had  1200  world-wide  par¬ 
ticipants  in  36  subject  areas.  Most  participants  are  U.S.  Army  officers,  though  use  is  open  to  per¬ 
sonnel  in  other  services  and  DoD  civilians  and  contractors.  Conferences  tend  to  focus  on 
professional  topics  of  common  interest.  The  network  is  used  for  both  formal  and  informal  sharing 
of  information,  experiences  and  expertise,  and  is  similar  to  the  bulletin  boards,  conferences,  and 
newsgroups  available  on  the  DDN  and  commercial  computer  networks.  The  network  allows  pro¬ 
fessionals  with  common  interests  to  communicate.  Thus,  for  example,  ar  artillery  officer  at  Ft.  Car- 
son,  CO,  can  pose  a  question  to  the  forum  and  be  answered  by  a  peer  at  Ft.  Lewis,  WA.  Stated 
applications  include  supplanting  of  face-to-face  meetings;  supplementing  of  other  forms  of  com¬ 
munication;  providing  assistance  in  coordinating  and  directing  geographically  dispersed  organiza¬ 
tions;  providing  a  medium  for  ongoing  discussions,  distribution  of  information;  creating  a  greater 
sense  of  community;  and  tailoring  the  communications  process  to  group  characteristics. 

The  AFN  is  administered  by  an  office  in  Washington,  DC.  Interested  parties  may  obtain  an  in¬ 
formation  packet  from  this  office  with  instructions  for  the  use  of  the  network  and  the  establishment 
of  discussion  groups.  Currently  the  Navy  has  no  equivalent  of  the  AFN,  although  the  communica¬ 
tion  infrastructure  for  such  a  conferencing  system  exists  in  the  DDN,  which  is  used  by  all  the  armed 
services  and  the  Army  Forum  network. 
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NPRDC  participated  briefly  in  the  Training  Research  discussion  group  of  the  AFN.  A  set  of 
instructions  was  obtained  from  Ft.  Monroe,  VA,  the  Army  provided  an  account,  and  a  link  to  the 
net  was  established  using  a  1200- baud  telephone  link  (the  highest  data  rate  supported).  The  com¬ 
munication  procedures  and  protocols  are  primitive  by  the  standards  of  today’s  high-speed  networks 
but  the  AFN  has  the  essential  functionality  to  enable  individuals  with  common  professional  inter¬ 
ests  to  network  their  knowledge  and  perform  group  problem  solving. 

Select  ISN  Testbed 

The  ISN  concept  was  applicable  in  many  different  instructional  environments.  Project  person¬ 
nel  conducted  an  informal,  small-scale  survey  of  Navy  instructional  environments  that  were 
considered  possible  candidates  for  the  ISN  early  in  the  project.  This  survey  included  Navy  techni¬ 
cal  schools.  Naval  Reserve  training,  the  Navy  correspondence  course  program,  college-  and  grad¬ 
uate-level  officer  training,  and  shipboard  and  pierside  training.  The  results  of  this  survey  are 
described  in  Simpson  (1990).  The  survey  eliminated  most  of  these  environments  as  possible  set¬ 
tings  for  the  ISN.  Prime  candidates  from  the  start  were  Navy  correspondence  course  program  and 
the  continuing  education  program  at  the  Naval  Postgraduate  School  (NPS).  Both  supported  inde¬ 
pendent  study  by  students,  which  seemed  suitable  for  a  system  such  as  the  ISN.  Navy  technical 
schools  were  rejected  after  school  personnel  said  that  their  students  were  highly  regimented  and 
lacked  the  time  and  initiative  to  use  computers  and  that  a  system  such  as  the  ISN  threatened  to  re¬ 
duce  the  amount  of  instructor- student  interaction.  The  survey  revealed  that  naval  reservists  have 
very  serious  training  problems,  that  some  type  of  distance  training/education  technology  could 
benefit  them  greatly,  but  that  the  ISN  was  not  the  suitable  technology  to  solve  those  problems;  a 
better  technology  was  videoteletraining  (see  Simpson,  1990;  Simpson,  Pugh,  &  Parchman,  in 
press).  The  same  conclusion  was  reached  regarding  shipboard  and  pierside  training. 

The  Navy  correspondence  course  program  appeared  to  be  a  good  candidate  for  the  ISN.  The 
program  has  a  large  number  of  students  and  courses  (approximately  60,000  students  in  580  cours¬ 
es)  and  students  engage  in  independent  study.  However  personnel  representing  this  program  saw 
little  need  for  the  ISN  as  the  course  completion  rate  overall  is  56  percent  and,  in  courses  required 
for  advancement  in  rating,  it  approaches  100%  (King,  1988). 

At  the  time  of  the  survey,  the  NPS  supported  a  continuing  education  program  that  enabled  non¬ 
resident  students  to  take  certain  courses  by  correspondence.  The  majority  of  students  were  active- 
duty  military  officers  stationed  many  miles  from  the  NPS.  The  most  common  motivation  for  taking 
courses  was  to  improve  their  academic  records  and  their  chances  of  gaining  admission  to  the  NPS 
or  other  highly  selective  schools  such  as  test  pilot  school.  The  courses  offered  were  correspondence 
versions  of  those  taught  at  the  NPS  and  were  technical  and  difficult;  roughly  half  of  the  students 
enrolled  in  college-level  mathematics  or  physics.  To  support  learning  on  a  remote  basis,  students 
were  supposed  to  recruit  a  local  tutor  knowledgeable  in  the  particular  subject  area.  The  attrition 
rate  in  these  courses  was  much  higher  than  for  Navy  correspondence  courses  generally.  According 
to  the  former  director  of  continuing  education,  of  the  4,500  students  who  signed  up  for  a  course 
each  year,  only  400  successfully  completed  the  course  (Zucker,  1988).  Possible  reasons  for  such 
a  high  attrition  rate  might  include  conflicts  with  work  commitments,  lack  of  adequate  tutoring,  or 
lack  of  the  peer  support  available  in  the  normal  classroom  learning  situation.The  ISN  seemed  to 
have  the  potential  to  overcome  the  tutoring  limitation  by  linking  the  students  to  an  expert  tutor  at 
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the  NPS.  It  also  had  the  potential  to  link  students  to  peers  and  thereby  provide  a  degree  of  peer  sup¬ 
port.  NPS  personnel  expressed  strong  interest  in  the  ISN  concept  and  offered  their  support  and  re¬ 
sources  for  developing  it  at  the  NPS. 

We  selected  the  continuing  education  program  at  the  NPS  as  a  testbed  for  the  ISN  for  several 
reasons.  Key  among  these  were  that  the  ISN  seemed  to  have  the  potential  to  reduce  the  severe  at¬ 
trition  problem,  school  personnel  were  supportive,  and  the  school  had  an  existing  computer  net¬ 
work  that  could  be  readily  tied  into  for  use  in  the  ISN. 

ISN  DESIGN  AND  DEVELOPMENT 


Overview 

The  ISN  was  conceived  as  a  prototype  electronic  mail  (e-mail)  system  whose  purpose  was  to 
link  geographically-remote  NPS  continuing-education  students  with  a  skilled  tutor  at  the  NPS.  The 
ISN  would  be  a  vehicle  for  communication  between  tutor  and  students,  enabling  the  student  to  ask 
questions  and  obtain  answers,  and  enabling  the  tutor  to  track  student  progress  and  provide  student 
guidance  and  motivation. 

Basic  Design  Considerations 

One  of  the  first  ISN  design  issues  was  that  computer  equipment  and  software  would  be  packed 
in  boxes  and  shipped  to  students  at  remote  locations,  where  they  would  be  expected  to  put  every¬ 
thing  together  and  make  it  work.  There  was  no  reason  to  expect  students  to  have  computer  skills 
and  it  would  not  be  feasible  to  train  students  face  to  face. 

Thus,  the  ISN  had  to  satisfy  certain  basic  design  requirements: 

1.  ISN  hardware  had  to  be  economical,  rugged,  and  easy  to  assemble. 

2.  User  documentation  had  to  be  simple  and  foolproof. 

3.  Users  would  have  to  be  able  to  assemble  and  operate  their  systems  and  diagnose  simple 
problems  using  only  the  documentation. 

4.  Users  without  computer  skills  would  be  able  to  use  the  ISN  successfully. 

Design  Principles 

Project  personnel  examined  several  MS-DOS  based  terminal  emulation  programs  in  the  hope 
of  finding  one  that  could  be  used  without  modification  in  the  ISN.  None  of  the  software  examined 
was  simple  enough  to  meet  the  third  design  requirement  (fisted  above).  The  terminal  programs  ex¬ 
amined  did  much  more  than  the  ISN  needed  to  do  and  required  considerable  user  computer  knowl¬ 
edge  to  use  successfully. 
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Consequently,  a  decision  was  made  to  have  a  contractor  develop  a  turnkey-style  program,  using 
the  authoring  language  with  the  “Procomm”  terminal  program  available  from  Datastorm  Technol¬ 
ogies.  Project  personnel  wrote  a  statement  of  work  describing  the  design.  The  statement  of  work 
and,  in  turn  in,  ISN  software  design  reflect  five  principles: 

1.  Simplicity.  The  program  has  the  minimum  number  of  functions  necessary  to  perform  its 
job  and  only  one  way  to  perform  each  action. 

2.  Visibility.  Menus  display  all  possible  program  options  on  the  screen. 

3.  Consistency.  Screens  are  laid  out  according  to  a  common  template,  and  information  is 
displayed  and  options  exercised  similarly  from  screen  to  screen. 

4.  Self-documentation.  The  program  displays  all  options  in  menus  and  provides  help  screens. 

5.  Unbreakability.  No  matter  what  the  user  does,  the  program  must  not  crash. 

These  design  principles  were  adopted  to  provide  a  starting  point  and  to  facilitate  communica¬ 
tion  with  the  contractor.  The  principles  are  idealistic  and,  when  they  were  adopted,  it  was  not  clear 
how  well  they  could  actually  be  satisfied.  Project  personnel  worked  closely  with  the  contractor  dur¬ 
ing  software  design  and  development.  They  tested  software,  had  representative  users  attempt  to  use 
the  system,  and  made  numerous  suggestions  for  changes. 

Hardware  Selection 

The  Army  Research  Institute  had  fielded  a  system  similar  to  the  ISN  during  the  Asynchronous 
Computer  Conferencing  project  and  recommended  the  use  of  IBM  PC  XT  compatible  Zenith- 1 84 
laptop  computers  and  ALPS  printers  which  were  more  rugged  and  easier  to  set  up  than  standard 
desktop  computers  (Hagman,  1988).  The  Z- 1 84  with  carrying  case  weighs  less  than  10  pounds  and 
includes  an  80-  by  24-character  LCD  display,  640  kbytes  of  RAM,  full  keyboard,  20  Mbyte  hard 
disk,  3.5-inch  floppy  disk,  and  built-in  1200-baud  Hayes-compatible  modem  and  printer  ports.  The 
compact  ALPS  printer  can  be  connected  to  it  with  a  single  cable,  and  a  phone  jack  can  be  plugged 
into  the  modem  port  in  the  side  of  the  computer.  The  telephone  line  is  connected  to  the  computer 
using  an  RJ-11  telephone  jack.  The  hardware,  at  GSA  prices,  costs  less  than  $1500  per  system. 
The  Z-184  and  ALPS  printer  were  selected  for  use  by  students  for  the  ISN  evaluation. 

System  Overview 

Figure  1  provides  an  overview  of  the  ISN  system.  The  system  consists  of  three  basic  elements: 
(1)  the  DDN  communications  network,  (2)  the  user’s  laptop  computer,  and  (3)  the  IBM  mainframe 
(host)  computer  at  the  NPS.  The  Department  of  Defense  developed  the  DDN  to  provide  high-speed 
communications  among  computers  throughout  the  United  States  and  overseas.  The  DDN  consists 
of  two  major  networks,  the  MILNET  and  the  ARPANET.  The  MILNET  handles  day-to-day  de¬ 
fense  data  communications  and  is  the  part  of  the  DDN  used  by  the  ISN  system.  As  shown  in 
Figure  1 ,  a  computer  at  the  NPS  in  Monterey,  CA  can  be  connected  to  computers  in  other  locations. 


9 


Figure  1.  Overview  of  Instructional  Support  Network. 


Through  the  DDN,  a  user  at  one  computer  can  connect  to  another  computer,  log  onto  it,  and  work 
on  the  second  computer.  Most  computers  on  the  network  are  connected  directly  to  the  DDN.  A  user 
also  can  connect  to  the  DDN  through  a  Terminal  Access  Controller  (TAC).  Typically,  a  modem 
connects  a  TAC  to  a  user’s  computer  or  terminal  through  telephone  lines.  The  computer  user  sends 
commands  to  the  modem  telling  it  to  dial  the  telephone  number  for  a  local  TAC.  Once  a  telephone 
connection  is  made  with  the  TAC,  the  user  logs  onto  the  TAC,  tells  the  TAC  to  connect  to  a  specific 
computer  on  the  network,  and  logs  onto  that  computer. 

The  host  computer  at  the  NPS  is  an  IBM  3033  running  the  VM/CMS  operating  system.  The 
computer  is  connected  directly  to  the  MILNET.  The  IBM  3033  is  a  multiuser  system  that  provides 
basic  file  handling,  data  processing,  and  communications  facilities.  The  VM/CMS  operating  sys¬ 
tem  also  provides  an  electronic  mail  facility.  The  IBM  mainframe  serves  as  the  hub  of  the  ISN  sys¬ 
tem.  All  messages  sent  between  students  and  tutor  pass  through  the  host  computer  and  are  stored 
on  it  until  retrieved  by  their  addressee. 

Using  the  ISN  is  analogous  to  using  a  post  office  box.  A  letter  writer  composes  a  letter  at  a  lap¬ 
top  computer,  takes  the  letter  to  the  post  office  (the  IBM  3033  computer),  and  puts  it  in  the  post 
office  box  for  the  recipient.  Later,  the  recipient  goes  to  the  post  office,  picks  up  the  letter  from  the 
post  office  box,  takes  it  home  (to  a  laptop  computer),  and  reads  it. 
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ISN  User’s  Software 


The  ISN  user’s  software  combines  a  text  editor,  terminal  program,  and  simple  mail  program. 
Students  use  the  software  to  compose  and  transmit  messages  to  their  tutor  over  the  DDN,  and  to 
receive  and  store  messages  from  the  tutor.  The  software  is  set  up  before  being  provided  to  the 
student.  The  user  must  enter  his  or  her  password  to  assure  authorization  to  use  the  ISN  but  does  not 
have  to  dial  a  phone  number,  turn  on  a  modem,  copy  files,  use  hierarchical  directories,  or  deal  with 
other  computer  arcana. 

The  software  also  provides  a  data  communications  handler  for  seeding  and  receiving  mail.  This 
handler  connects  the  user’s  computer  to  the  MILNET  and  the  host  computer  at  the  NPS  and  also 
handles  the  transfer  of  mail  files  between  the  computers.  The  software  provides  commands  for 
managing  the  mail  files  such  as  deleting  and  printing  messages. 

The  software  combines  custom  and  commercial  software  packages.  The  custom  software  man¬ 
ages  the  system,  controls  other  software  (CSE  text  editor,  Procomm  communications  software), 
provides  the  menu  and  command  line  interface,  connects  each  computer  to  the  IBM  3033  at  NPS 
through  the  MILNET,  and  automatically  transfers  message  files  between  the  computers.  The  cus¬ 
tom  software  is  a  combination  of  executable  programs,  batch  files,  and  text  files.  It  includes  several 
configuration  programs,  which  are  used  to  set  up  or  modify  each  laptop  computer  for  the  particular 
user. 

The  CSE  text  editor,  which  is  used  to  read  and  edit  messages,  is  a  public-domain  program  de¬ 
veloped  at  the  Colorado  School  of  Mines.  The  editor,  though  simple,  permits  the  keyboard  to  be 
customized.  The  ISN  implementation  purposely  limits  available  editing  functions  to  the  very  min¬ 
imum  such  as  insert,  delete,  overwrite. 

Procomm  is  a  popular  and  fairly  powerful  data  communications  package  developed  by  Datas- 
torm  Technologies,  Inc.  Procomm  handles  data  communications  between  different  computers, 
transfers  files  between  machines  using  a  variety  of  file  transfer  protocols  (Kermit,  developed  at  Co¬ 
lumbia  University,  is  used  in  the  ISN  implementation),  and  runs  command  scripts.  Command 
scripts  were  used  in  the  ISN  to  transfer  files  to  and  from  the  IBM  computer  at  NPS. 

Figure  2  shows  the  organization  of  the  ISN  program.  The  seven  ISN  program  operations  are 
capitalized-TRANSMIT,  RECEIVE,  READ,  COMPOSE,  DELETE,  PRINT,  QUIT.  All  opera¬ 
tions  except  QUIT  operate  on  the  Message  File. 

TRANSMIT  and  RECEIVE  are  both  “on-line”  operations.  They  require  only  the  Z-184  to  be 
connected  to  a  phone  line.  TRANSMIT  transmits  messages  from  the  Message  File  to  the  NPS. 
When  selected,  TRANSMIT  automatically  calls  the  NPS  computer  on  the  phone,  sends  the  mes¬ 
sage  to  the  NPS  computer,  and  hangs  up  the  phone.  RECEIVE  receives  messages  from  the  NPS 
and  stores  them  in  the  Message  File.  RECEIVE  is  typically  used  first,  after  starting  the  program. 
When  selected,  RECEIVE  automatically  calls  the  NPS  computer  on  the  phone,  signs  on,  collects 
new  messages,  puts  them  in  the  Message  File,  signs  off,  and  hangs  up  the  phone. 

The  remaining  operations  are  performed  “off  line.”  READ  displays  new  messages  received 
from  the  tutor.  COMPOSE  allows  the  student  to  type  a  message  with  a  very  simple  text  editor  and 
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[.Organization  of  Instructional  Support  Network  program. 


store  it  in  the  Message  File.  DELETE  allows  the  student  to  discard  messages  from  the  Message 
File.  PRINT  allows  the  student  to  print  hard  copies  of  messages.  QUIT  ends  the  program. 

The  program’s  main  menu  screen  is  shown  in  Figure  3.  When  an  option  is  selected,  a  secondary 
screen  is  presented.  All  secondary  screens  look  and  behave  similarly.  There  are  no  third-level 
screens,  typed-in  commands,  or  setup  displays. 


Instructional  Support  Network  (ISN) 


Main  Menu 


»  RECEIVE;  Receive  mail  messages  form  your  tutor  at  NPS. 


READ:  Read  mail  messages  received  from  your  tutor  at  NPS. 
COMPOSE:  Compose  mail  messages  to  your  tutor  at  NPS. 
TRANSMIT:  Send  mail  messages  to  your  tutor  at  NPS. 
DELETE:  Select  and  delete  mail  messages. 

PRINT:  Select  and  print  mail  messages. 

QUIT:  Leave  ISN  program  and  return  to  DOS. 


Use  ^  and  ^  to  select  menu  option.  Then  press  the  ENTER  key. 


Figure  3.  Instructional  Support  Network  main  menu  screen  for  student  software. 

ISN  User  Documentation 

A  decision  was  made  to  provide  users  with  the  minimum  amount  of  documentation  necessary 
to  assemble  their  systems,  operate  them,  and  resolve  any  possible  problems  (Simpson  &  Pugh, 
1989).  The  guide  consists  of  brief  overview  material  (“Welcome,”  “About  this  Guide,”  “Preface”) 
and  four  content  chapters: 

1 .  Getting  Started-inventorying,  assembling,  and  testing  hardware. 

2.  How  to  Use  the  ISN  Program-a  tutorial  on  how  to  use  the  program. 

3.  ISN  Communication  Conventions— communication  style,  rules  for  communication,  how  to 
compose  mathematical  expressions. 
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4.  Handling  Problems--What  to  do  when  something  doesn’t  work. 

The  guide  is  as  short  and  simple  as  possible.  Chapter  1,  which  tells  how  to  assemble  hardware, 
contains  numerous  illustrations;  Figure  4  shows  a  typical  page.  Chapter  3  presents  brief  step-by- 
step  instructions.  Chapters  3  and  4  are  short  and  organized  like  glossaries. 

It  was  hoped  that  users  would  read  the  guide,  but  assumed  that  they  might  not.  Consequently, 
users  were  provided  with  a  “hot  line”  phone  number  to  call  in  the  event  of  problems  and  the  soft¬ 
ware  was  written  with  extensive  prompts  and  help  messages  to  be  internally  “self-documenting.” 

Software  Testing 

Project  personnel  tested  ISN  software  continually  as  it  was  being  developed  by  the  software 
contractor.  This  testing  commenced  when  the  first  software  modules  were  ready  for  use  and  con¬ 
tinued  throughout  software  development,  a  period  of  approximately  six  months.  The  tests  were 
performed  to  identify  and  correct  software  bugs  and  to  assure  that  the  software  met  the  design  prin¬ 
ciples  of  simplicity,  visibility,  consistency,  and  self-documentation.  Software  tests  were  designed 
to  exercise  every  probable  user  action  as  well  as  less  probable  but  possible  user  actions.  Testers 
attempted  to  misuse  the  software  as  well  as  to  use  it  as  intended;  various  errors  were  intentionally 
made  in  order  to  ensure  that  the  system  would  not  crash  (the  unbreakability  principle). 

After  preliminary  software  testing,  the  ISN  was  installed  by  the  contractor  and  used  in  a  beta 
test  environment  by  both  the  contractor  and  project  personnel.  The  ISN  was  used  for  within-office 
communication  for  about  two  months  before  the  field  test  began.  The  beta  test  allowed  project  per¬ 
sonnel  to  evaluate  the  capability  of  the  ISN  to  meet  the  needs  of  users.  Beta  testing  uncovered  sev¬ 
eral  problems  not  evident  during  initial  testing;  e.g.,  the  NPS  mainframe  purged  messages  at  three- 
day  intervals,  resulting  in  lost  messages.  These  problems  were  corrected  before  conducting  the  for¬ 
mative  evaluation  and  field  test. 

ISN  EVALUATION  METHODOLOGY 


Objectives 

The  ISN  evaluation  had  three  main  objectives: 

1.  Determine  how  easy  it  was  to  learn  to  operate  and  use  the  ISN. 

2.  Document  the  support  requirements  and  problems  associated  with  installing  and 
maintaining  a  system  such  as  the  ISN. 

3.  Determine  the  impact  of  the  ISN  upon  student  attrition,  performance,  and  attitudes. 

The  context  of  the  evaluation  was  a  sequence  of  NPS  Continuing  Education  (CE)  calculus 
courses. 

The  first  and  second  objectives  focus  on  process  and  the  third  focuses  on  product. 
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7 .  Locate  the  grounding  clip  on  the  printer 
end  of  the  cable. 

8.  Connect  the  grounding  clip  to  the 
Phillips  head  screw  above  and  to  the 
left  of  the  cable  connecter  on  the  back  of 
the  printer  (see  Figure  1-8). 

9.  Connect  the  power  cord  to  the  back  of  the 
printer. 

10.  Plug  the  power  cord  into  a  grounded 
outlet. 

11.  If  a  floppy  disk  is  in  the  drive  of  your  Z- 
184,  remove  the  disk. 

12.  Turn  on  your  Z- 184. 

13.  Turn  on  your  printer. 

Figure  4.  Typical  page  from  Instructional  Support  Network  User’s  Guide. 
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Formative  Evaluation 


A  formative  evaluation  of  the  ISN  was  conducted  to  ensure  that  the  aspects  of  the  ISN  that  we 
could  control— hardware,  software,  and  documentation— were  sufficiently  refined  for  release  to  the 
ISN’s  intended  users.  Though  we  had  made  every  effort  to  test  and  refine  the  ISN,  we  were  sure 
that  some  of  its  shortcomings  would  only  become  evident  once  it  was  in  the  hands  of  actual  users. 

We  solicited  test  subjects  from  NPRDC  personnel  by  running  a  notice  in  the  NPRDC  weekly 
newsletter.  The  notice  requested  personnel  with  limited  computer  experience  to  act  as  test  subjects. 
The  incentive  to  participate  was  to  gain  some  experience  with  the  Z-184  computer.  The  five 
NPRDC  volunteers  had  little  or  no  computer  experience. 

Each  subject  was  shown  to  a  room  and  provided  with  a  copy  of  the  ISN  User’s  Guide,  a  Phillips 
head  screwdriver,  and  the  cartons  containing  the  ISN  equipment.  A  researcher  instructed  each  sub¬ 
ject  to  follow  the  directions  in  the  user’s  guide  to  unpack  and  set  up  the  ISN,  and  to  use  it  to  send 
and  receive  a  message.  The  researcher  then  left  the  room  and  waited  for  the  subject  to  complete  the 
task  alone. 

Process  Measures 

The  following  forms  of  process  data  (relating  to  evaluation  objectives  1  and  2)  were  collected 
during  the  ISN  field  test: 

Student  message  traffic.  All  student  and  tutor  messages  were  sent  both  to  the  addressee  and 
to  a  file  accessible  to  researchers.  These  messages  provided  a  historical  record  of  all  student- 
tutor  communications.  It  was  anticipated  that  the  message  traffic  would  eventually  be  analyzed 
to  develop  a  number  of  descriptive  measures  relating  to  message-generation  rate  and  message 
content  (e.g.,  administrative,  technical,  motivational,  etc.). 

Support  requirements.  The  ISN  evaluation  required  the  support  of  a  tutor,  software  engineer, 
hot-line  person,  and  programmer  in  the  computer  center  at  NPS.  We  recorded  the  amount  of 
time  these  individuals  spent  per  week  supporting  the  ISN  during  the  field  test. 

Tutor  comments.  Researchers  and  the  tutor  met  before,  during,  and  after  the  field  trial.  Notes 
were  kept  for  all  these  meetings.  A  formal  debrief  interview  was  conducted  with  the  tutor  and 
his  supervisor  at  the  end  of  the  field  trial.  In  addition,  the  tutor  provided  written  comments  on 
strengths  and  weaknesses  of  the  ISN  and  suggestions  for  improvements. 

Contractor  technical  problem  log.  The  software  contractor  recorded  all  problems 
encountered  during  software  development  and  field  testing.  These  problems  were  described  in 
a  technical  report  that  the  contractor  submitted  at  the  conclusion  of  the  ISN  software 
development  (Newbury,  1989).  Key  information  from  this  report  is  summarized  in  the  Findings 
section  of  this  report. 

Formal  Evaluation 

The  formal  evaluation  of  the  ISN  focused  on  evalua  :on  objective  3. 
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Research  Design 

A  two-group  design  was  used  with  an  experimental  group  (participants  in  ISN)  and  control 
group  (conventional  instruction). 

Subjects  in  the  experimental  group  were  provided  with  the  ISN  and  received  tutoring  via  the 
ISN  from  a  tutor  at  the  NPS;  these  subjects  were  not  permitted  to  receive  help  from  a  local  tutor. 
Subjects  in  the  control  group  were  not  provided  with  the  ISN  and  received  tutoring  in  the  conven¬ 
tional  way  from  a  local  tutor  whom  they  had  to  recruit.  One  of  the  shortcomings  of  the  CE  system 
is  that  students  sometimes  had  difficulty  finding  a  qualified  tutor  or  were  reluctant  to  make  de¬ 
mands  upon  their  tutor’s  time.  As  a  consequence,  some  students  either  did  not  work  with  a  tutor  or 
did  not  take  full  advantage  of  the  tutor’s  skills.  It  was  anticipated  that  the  ISN  would  overcome  this 
limitation  because  a  highly-qualified  tutor  would  be  available  with  minimum  effort  from  the  stu¬ 
dent.  Based  on  this  analysis,  it  seemed  a  reasonable  hypothesis  that  the  ISN  would  positively  affect 
students  in  the  experimental  group. 

Except  for  the  manner  in  which  tutoring  was  provided,  all  subjects  received  the  same  training 
and  testing  materials  and  followed  the  same  procedures.  Students  were  enrolled  in  NPS  CE  math¬ 
ematics  courses  numbered  MAI  133  through  MA  1140.  This  series  of  beginning  calculus  courses 
covers  the  same  material  as  the  first  year  of  college  calculus.  Students  in  both  groups  were  taking 
courses  at  various  levels;  i.e.,  not  all  students  in  both  groups  were  taking  exactly  the  same  course. 
Course  materials  consisted  of  a  student  textbook,  guidebook,  and  final  examination. 

Subjects 

It  was  originally  planned  to  use  a  total  of  40  subjects,  with  20  in  each  group.  All  subjects  were 
participants  in  the  NPS  CE  calculus  course  sequence.  The  majority  of  subjects  were  active  duty 
military  personnel  with  a  rank  equivalent  to  or  lower  than  Navy  lieutenant.  Some  DoD  civilians 
also  participated  in  the  study.  The  NPS  CE  office  provided  researchers  with  a  list  of  names  and  tele¬ 
phone  numbers  of  personnel  who  had  enrolled  in  one  of  the  CE  calculus  courses.  A  researcher  con¬ 
tacted  each  enrollee  by  telephone.  Subjects  were  randomly  assigned  to  a  group  (experimental  or 
control)  without  regard  to  their  educational  background,  mathematical  ability,  or  other  variables. 
Subjects  who  either  expressed  unwillingness  to  use  the  ISN  or  did  not  have  convenient  (i.e.,  toll 
free)  access  to  a  TAC  for  communicating  over  the  MELNET  were  excluded  from  the  experimental 
group.  Two  subjects  refused  to  participate  in  the  experimental  group  because  they  had  a  highly 
qualified  local  tutor. 

All  subjects  in  the  experimental  group  were  provided  with  Z-184  computers  and  ALPS  printers 
and  ISN  software  described  in  greater  detail  in  the  ISN  Design  and  Development  section  of  this 
report. 

Data  Collection 

It  was  originally  planned  to  use  the  methods  and  collect  the  data  as  described  below: 
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Student  Academic  Profile  Code.  This  composite  score,  computed  by  the  NPS,  reflects  grade 
point  average  and  level  of  achievement  in  engineering-  and  science-related  courses.  This 
measure  was  to  be  obtained  from  the  NPS  CE  department. 

Pre-course  background  interview.  The  questionnaire  used  during  the  pre-course  telephone 
interviews  of  students  contained  questions  concerning  the  student’s  computer  background, 
willingness  to  participate  as  a  test  subject,  and  a  ~ess  to  MILNET.  It  was  used  both  to  gather 
student  background  information  and  assign  subjects  to  the  experimental  or  control  group. 

Student  final  examination  scores.  These  scores  were  to  be  obtained  from  the  NPS  CE 
department. 

Student  debrief  interview.  The  questionnaire  used  during  post-course  telephone  interviews  of 
students  contained  questions  concerning  students  experiences  in  and  attitudes  toward  the 
course. 

Research  Variables 

Subjects  in  the  experimental  group  participated  in  the  ISN  and  those  in  the  control  group  re¬ 
ceived  conventional  tutoring.  The  independent  variable  is  type  of  tutoring;  i.e.,  ISN  tutoring  versus 
traditional  tutoring. 

Three  dependent  variables  were  chosen: 

Attrition.  Since  NTS  CE  instruction  has  histo  '.ally  experienced  a  high  attrition  rate,  attrition 
rate  was  an  important  measure  of  ISN  eiiectiveness.  The  field  trial  period  was  limited  to  several 
months,  allowing  as  few  as  four  months  for  a  particular  student  to  participate;  historically, 
students  have  sometimes  takv.  n  up  to  ?  vea*-  to  complete  a  course.  Hence,  we  developed  an 
operational  definition  of  course  completion  for  use  for  a  shorter  time  window.  We  defined 
course  completion  as  the  percentage  of  students  in  either  group  (experimental  or  control)  who 
had  completed  the  course  in  eight  weeks.  The  eight  weeks  were  measured  from  the  date  the 
course  materials  were  sent  to  the  date  the  course  final  examination  was  received  by  NPS. 

Performance.  The  score  on  the  course’s  final  exam  represents  student  performance. 

Attitudes.  Several  measures  were  obtained  from  students  at  the  termination  of  the  project 
regarding  their  attitudes  toward  the  course,  method  of  instruction,  and  quality  of  instruction. 

The  evaluation  plan  called  for  gathering  data  on  control  variables  relating  to  student  individual 
differences.  Student  scholastic  ability  was  to  be  determined  from  student  Academic  Profile  Code. 
Tek  phone  interviews  of  students  were  planned  to  obtain  data  concerning  student  workload,  sched¬ 
ule  conflicts,  interest  in  course  material,  career  goals,  and  student  mathematical  ability.  Students 
who  participated  in  the  ISN  were  also  asked  to  report  on  various  problems  in  using  the  ISN. 

Changes  to  Formal  Evaluation  Methodology 

The  evaluation  plan  was  written  during  the  fall  of  1988.  When  the  ISN  field  test  commenced 
in  mid-February  1989,  the  NPS  CE  department  began  to  provide  NPRDC  with  the  names  of  course 
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enrollees  at  the  rate  of  about  two  per  week.  After  about  six  weeks,  the  NPS  CE  department  in¬ 
formed  NPRDC  that  the  CE  program  was  under  scrutiny  by  NPS  management  and  was  in  danger 
of  being  terminated;  no  further  names  of  students  were  provided.  After  several  weeks  of  uncertain¬ 
ty,  the  CE  program  was  terminated.  NPS  stated  that  it  would  support  current  enrollees  in  the  pro¬ 
gram,  but  would  not  enroll  new  students.  However,  attrition  in  the  small  staff  of  the  CE  department 
and  failure  to  replace  the  loss  severely  reduced  the  ability  of  the  department  to  support  students  and 
provide  us  with  data.  Termination  of  the  CE  program  had  the  following  effects  on  the  formal  eval¬ 
uation: 

1.  The  number  of  students  in  the  experimental  group  was  reduced  from  20  (planned)  to  10 
(the  number  recruited  when  the  program  was  terminated).  We  obtained  the  names  of  20 
students  for  the  control  group.  Consequently,  the  number  of  students  was  10  ISN  and  20 
controls,  for  a  total  of  30. 

2.  The  NPS  was  unable  to  provide  student  final  examination  scores;  this  precluded  the  use  of 
this  variable  as  a  measure  of  student  achievement. 

3.  The  NPS  was  unable  to  provide  student  Academic  Profile  Codes;  this  precluded  use  of  this 
variable  as  a  measure  of  student  scholastic  ability. 

The  unanticipated  developments  in  the  CE  program  crippled  the  formal  evaluation  in  two  ways. 
First,  the  reduction  in  subject  population  size  reduced  the  likelihood  that  statistically  significant  re¬ 
sults  could  be  obtained  in  comparisons  between  experimental  and  control  groups.  Second,  the  in¬ 
ability  of  the  NPS  to  provide  objective  measures  of  test  performance  and  academic  background 
forced  us  to  rely  more  oi.  process-type  measures  obtained  from  questionnaires  than  on  objective, 
product-type  measures. 


FINDINGS 


Overview 

This  section  describes  the  ISN  evaluation  in  terms  of  the  formative  evaluation,  process  mea¬ 
sures,  and  formal  evaluation. 

Formative  Evaluation 

All  subjects  were  able  to  unpack  their  system,  set  it  up,  get  it  running,  receive  a  message,  print 
a  message,  compose  a  message,  and  send  a  message  within  less  than  two  hours.  During  post-per¬ 
formance  interviews,  all  subjects  stated  that  they  had  found  the  directions  in  the  user’s  guide  easy 
to  follow  and  the  software  easy  to  use.  Subjects  offered  several  suggestions  for  improving  the 
guide;  many  of  these  suggestions  were  eventually  incorporated  in  the  revised  vusion  of  the  user’s 
guide. 
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Process  Measures 


Student  Message  Traffic 

All  student  and  tutor  messages  were  forwarded  to  a  computer  file  at  NPRDC  for  later  analysis 
to  determine  message  content  and  origin.  The  content  of  each  message  was  classified  as  adminis¬ 
trative  (nontechnical,  relating  to  course  procedures,  schedules,  rules),  technical  (relating  to  course 
content),  or  motivational  (tutor  reminders,  encouragements,  and  warnings  about  class  participa¬ 
tion).  The  origin  of  each  message  was  classified  as  student  or  tutor.  Table  1  summarizes  the  mes¬ 
sage  content  data  for  the  225  messages.  The  majority  (72.9%)  were  administrative,  17.3  percent 
were  technical,  and  9.8  percent  were  motivational.  Considering  that  the  ISN  was  intended  to  be  a 
tutoring  system,  the  percentage  of  technical  communication  is  surprisingly  low.  Three  of  the  four 
students  who  completed  the  course  (students  1,  5,  and  10)  sent  almost  no  technical  messages.  Stu¬ 
dents  3  and  4  account  for  the  majority  of  all  technical  communication  but  only  student  3  completed 
the  course. 


Table  1 


Message  Content  by  Individual  Student  and  Overall 


Student 

Message  Origin 

Administrative 

Technical 

Motivational 

Total 

1* 

17 

0 

4 

21 

2 

14 

2 

4 

20 

3* 

13 

14 

1 

28 

4 

25 

13 

2 

40 

5* 

21 

0 

1 

22 

6 

20 

3 

1 

24 

7 

17 

6 

1 

24 

8 

2 

0 

2 

4 

9 

9 

0 

3 

12 

10* 

26 

1 

3 

30 

Totals 

164 

3« 

22 

225 

Percent 

72.9 

17.3 

9.8 

100.0 

♦Students  who  finished  the  course. 


Nontechnical  messages  may  have  been  of  motivational  value  by  linking  otherwise  isolated  stu¬ 
dents  to  someone  interested  in  their  success. 
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Table  2  summarizes  the  message  origin  data.  Overall,  students  initiated  40.9  percent  of  all  mes¬ 
sages;  the  tutor  generated  the  majority  of  messages.  One  might  expect  that  more  successful  stu¬ 
dents  would  tend  to  initiate  a  higher  percentage  of  their  messages  but  the  data  do  not  support  this 
idea. 


Table  2 

Message  Origin  by  Individual  Student  and  Overall 


Message  Origin 


Percent  Student 


Student 

Student 

Tutor 

Total 

Generated 

1* 

7 

14 

21 

33.3 

2 

6 

14 

20 

30.0 

3* 

13 

15 

28 

46.4 

4 

23 

17 

40 

57.5 

5* 

7 

15 

22 

31.8 

6 

10 

14 

24 

41.7 

7 

10 

14 

24 

41.7 

8 

0 

4 

4 

0.0 

9 

3 

9 

12 

25.0 

10* 

13 

17 

30 

43.4 

Totals 

92 

133 

225 

40.9 

♦Students  who  finished  the  course. 


ISN  Support  Requirements 

The  ISN  required  several  kinds  of  support.  Support  personnel  included  a  full-time  tutor  and  a 
pan-time  software  engineer  who  monitored  the  network  to  identify,  diagnose,  and  correct  technical 
problems  and  provided  technical  suppon.  In  addition,  a  researcher  was  available  to  provide  hot¬ 
line  support  to  students  who  were  experiencing  difficulties,  and  personnel  in  the  NPS  computing 
center  provided  occasional  support  to  the  ISN.  The  amount  of  support  was  greater  when  the  system 
was  being  set  up  than  after  its  operation  had  reached  a  steady  state  and  it  was  running  smoothly. 
After  that  point  was  reached,  the  technical  support  by  the  software  engineer  averaged  1  or  2  hours 
per  week,  hot-line  support  about  30  minutes  per  week,  and  computer  center  support  about  30  min¬ 
utes  per  week.  The  tutor  estimated  that  he  could  have  tutored  40  students  with  the  ISN  (and  perhaps 
more),  since  he  was  able  to  tutor  10  students  in  less  than  10  hours  per  week. 
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T\itor  Comments 


During  the  final  month  of  the  field  test,  two  project  researchers  conducted  an  open-ended  inter¬ 
view  of  the  NPS  tutor.  They  ashed  a  series  of  questions  concerning  the  tutor’s  perceptions  of  1SN 
strengths  and  weaknesses  and  suggestions  for  improvements.  In  addition,  they  asked  the  tutor  for 
written  comments  on  the  ISN.The  written  and  oral  comments,  which  followed  closely,  are  summa¬ 
rized  below:. 

The  tutor  stated  that  the  ISN  was  a  “great”  tutoring  tool.  It  was  easy  to  learn  to  operate  and  use, 
and  both  students  and  tutor  could  operate  it  effectively  in  a  short  time. 

The  tutor  identified  several  problems  during  the  field  test: 

1.  Slow  student  progress  with  lessons.  This  is  not  a  problem  with  the  ISN,  but  with  student 
performance  in  CE  programs  generally.  The  tutor  believed  that  most  students  tended  to 
progress  slowly  because  of  conflicts  with  their  work  duties,  home  life,  or  other  interests. 

2.  Lack  of  communication  by  students.  The  ISN  “contract”  (the  terms  under  which 
students  were  lent  their  computer)  required  that  each  student  communicate  at  least  once  per 
week.  Most  did  not  communicate  this  often  or  communicated  in  a  very  cursory  fashion. 

3.  NPS  host  computer  problems  that  precluded  student-tutor  communication.  At  various 
times,  the  IBM  host  computer  rejected  or  lost  ISN  messages  and  the  communicators  could 
not  tell  that  their  messages  were  not  getting  through.  This  problem  was  caused  by 
unanticipated  side  effects  of  IBM  software  modifications  and  was  only  detected  after  long 
periods  of  noncommunication. 

4.  DDN  problems  that  precluded  student-tutor  communication.  The  DDN,  at  various 
times,  disrupted  communication.  The  primary  cause  was  unreliable  TAC  phone  lines  that 
would  reject  telephone  communications. 

5.  Inability  (by  tutor)  to  diagnose  cause  of  student  noncommunication.  There  was  no  easy 
way  to  tell  if  student  disinterest,  difficulty  operating  the  ISN  computer  or  software,  host 
computer  problems,  or  DDN  problems  caused  the  lack  of  communication. 

The  tutor  made  several  suggestions  for  improving  the  ISN: 

1 .  Provide  students  with  a  syllabus  containing  a  suggested  schedule  for  progress  through  their 
course. 

2.  Work  with  students’  work  supervisors  to  allow  them  to  “ease  off’  on  their  duties  to  provide 
more  time  for  their  studies. 

3.  Improve  the  ISN  software  by  providing  additional  capabilities  such  as  improved  graphics 
capabilities,  a  hypertext-style  subject  matter  database,  and  improved  text  editor. 

The  tutor  worked  under  the  supervision  of  a  senior  mathematics  professor  who  oversaw  ISN 
tutoring  at  NPS  but  did  not  tutor  students  with  the  ISN.  The  supervising  professor  had  developed 
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content  for  several  CE  mathematics  courses  and  had  several  years  of  experience  with  the  CE  pro¬ 
gram.  Though  he  was  not  directly  involved  in  ISN  tutoring,  his  experience  enabled  him  to  put  the 
ISN  into  the  context  of  conventional  tutoring  and  to  make  a  comparison  between  the  two.  The  pro¬ 
fessor  was  interviewed  shortly  after  the  tutor,  using  the  same  protocol.  The  professor  remarked  that 
the  biggest  problem  with  CE  courses  was  to  get  students  to  complete  them  According  to  the  pro¬ 
fessor,  the  school  had  no  system  to  follow  up  on  students,  but  the  ISN  provided  at  least  a  partial 
answer  to  this  problem.  He  remarked,  “My  impression  is  that  ISN  has  very  good  effects  on  students 
staying  in  the  program.”  He  added  that  the  system  of  instruction  used  in  CE  courses  was  based  on 
students  working  with  tutors  who  possessed  subject  matter  knowledge  but  that  many  students  were 
unable  to  find  suitable  tutors;  the  ISN  compensated  for  this  shortcoming. 

Technical  Problems 

This  section  summarizes  the  technical  problems  that  NPRDC  and  its  software  development 
contractor  encountered  with  the  ISN  while  it  was  in  use,  described  in  detail  in  Newbury  (1989). 

The  ISN  is  a  complex  arrangement  of  computers,  software,  and  long-distance  networks.  Sys¬ 
tem  operability  depends  on  all  components  working  properly;  the  failure  of  any  single  component 
usually  compromises  the  entire  system.  While  the  ISN  was  in  use,  nearly  every  component  in  it 
failed  at  least  once.  Most  failures  were  minor  and  were  quickly  corrected,  but  a  few  affected  the 
ISN  for  several  weeks  and  required  software  modifications. 

Table  3  summarizes  the  types  of  failures  that  occurred  while  the  system  was  in  use.  Not  all  the 
documented  problems  were  reported  by  the  students.  Some  reported  problems  were  noticed  by  the 
researchers  at  NPRDC  and  were  corrected  before  they  affected  the  operation  of  the  student  sys¬ 
tems. 


Table  3 

Summary  of  Technical  Problems 
(from  Newbury,  1989) 


System  Component 

Number  of  Failures 

Laptop  Hardware 

2 

Custom  Laptop  Software 

3 

Commercial  Laptop  Software 

0 

Improper  Laptop  Configuration 

2 

NPS  IBM  Hardware 

8 

NPS  IBM  Software 

8 

MILNET  Hardware 

6 

TAC  Hardware 

5 

Telephone  Lines 

1 

Total 

35 
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Laptop  hardware  failures.  Failures  of  the  Z-184  computers  were  rare  and  were  usually  cor¬ 
rected  by  replacing  the  broken  equipment.  In  one  case  the  power  supply  for  the  computer  failed. 
In  another  case,  there  were  undiagnosed  problems  with  the  computer  and  it  was  replaced.  The  big¬ 
gest  problem  with  hardware  failures  was  that  the  equipment  had  to  be  packed  and  sent  to  San  Di¬ 
ego,  and  new  equipment  had  to  be  shipped  to  the  student.  While  this  was  happening,  the  student 
could  not  use  the  system. 

Custom  laptop  software  failures.  Early  versions  of  the  custom  laptop  software  contained 
some  bugs  that  were  found  by  NPRDC  and  its  software  development  contractor  shortly  after  the 
ISN  became  operational.  Two  of  the  configuration  programs  contained  bugs  that  caused  data  to  be 
lost  and  another  bug  in  the  software  caused  it  to  crash;  however,  these  problems  were  corrected 
before  the  software  was  shipped  to  students. 

Commercial  laptop  software  problems.  Some  problems  were  encountered  with  the  CSE  text 
editor  and  the  Procomm  communications  software  during  ISN  development.  Our  programmers 
lacked  the  source  code  for  these  packages  and  could  not  modify  the  underlying  programs.  Conse¬ 
quently  they  had  to  find  ways  to  work  around  certain  program  deficiencies.  For  example,  they  im¬ 
provised  ways  to  disable  the  keyboard  while  Procomm  was  running  (which  would  break 
communication)  and  to  prevent  users  from  typing  past  the  80th  column  when  using  the  text  editor 
(which  would  shift  the  display  laterally).  Both  commercial  software  packages  were  very  reliable 
and  there  were  no  reported  failures  of  either  one. 

Improper  laptop  configuration.  In  two  cases,  the  laptop  software  was  improperly  configured 
before  systems  were  shipped  to  students.  In  both  cases,  the  problem  was  identified  with  a  hot  line 
telephone  conversation  and  NPRDC  personnel  provided  the  users  with  instructions  for  correcting 
their  problem. 

NPS  IBM  hardware  failures.  Host  IBM  hardware  failures  include  both  hardware  failures, 
which  were  infrequent,  and  when  the  host  was  down  for  planned  maintenance  or  due  to  operating 
system  crashes.  These  failures  occurred  often  enough  that  students  could  not  transmit  or  receive 
messages  when  they  wanted.  If  a  student  attempted  to  send  or  receive  a  message  when  the  host 
was  down,  ISN  software  sent  a  message  instructing  the  student  to  transmit  the  message  later. 

All  multiuser  systems  require  some  down  time  for  maintenance  and  to  handle  the  occasional 
hardware  and  software  crashes.  The  staff  at  NPS  reported  that  their  system  operates  99  percent  of 
the  time.  However,  at  least  five  times  during  June  1989,  the  host  computer  was  not  running  during 
normal  working  hours.  While  the  host  IBM  computer  appeared  to  be  down  more  often  than  other 
computer  systems,  it  was  not  possible  to  determine  if  the  down  time  was  excessive. 

NPS  IBM  software  failures.  Host  IBM  software  failures  consisted  of  VM/CMS  operating 
system  failures  and  MILNET  software  failures. 

VM/CMS  operating  system  problems  caused  two  failures.  The  first  was  caused  by  a  shortage 
of  disk  space  on  the  host,  which  the  NPS  computer  center  staff  dealt  with  by  deleting  all  messages 
more  than  three  days  old.  Since  students  did  not  necessarily  read  their  messages  every  three  days, 
this  resulted  in  the  unfortunate  loss  of  many  unread  messages.  When  the  oversight  was  realized. 
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NPS  staff  exempted  student  accounts  from  the  automatic  message  deletion  requirement  The  sec¬ 
ond  failure  occurred  when  the  host  computer’s  software  started  returning  all  messages  to  the  send¬ 
er.  The  problem  lasted  several  days,  and  was  eventually  corrected  by  the  NPS  staff. 

Failures  in  the  MILNET  networking  software  on  the  host  computer  resulted  in  several  com¬ 
munication  interruptions.  The  VM/CMS  operating  system  does  not  provide  strong  support  of  MIL- 
NET  communications  and  the  problems  experienced  are  common  to  the  family  of  computers 
running  this  operating  system.  Eventually  all  of  these  problems  were  corrected,  but,  while  present, 
they  had  serious  effects.  For  several  weeks,  NPRDC  researchers  were  unable  to  connect  to  the  host 
computer  at  NPS  through  the  MILNET  because  of  networking  software  problems  on  the  host.  A 
more  serious  problem  involved  the  routing  of  message  copies  from  the  host  to  the  message  archive 
on  the  NPRDC  VAX  computer.  NPS  computer  center  personnel  discovered  a  problem  in  their  MIL- 
NET  software  and  changed  the  software  to  route  all  outgoing  messages  through  the  BITNET  net¬ 
work  to  a  computer  on  the  east  coast,  where  the  messages  were  transferred  to  the  MILNET  for  final 
delivery.  NPS  made  this  software  change  without  notifying  us  and  the  ISN  began  to  experience 
several  problems  until  we  were  able  to  modify  our  software  to  deal  with  the  new  routing  require¬ 
ments. 

MILNET  hardware  failures.  All  reported  failures  of  the  MILNET  hardware  were  traced  to 
a  failure  of  MILNET  hardware  in  San  Diego.  These  problems  affected  only  students  in  the  San 
Diego  area.  Identifying  and  tracing  the  source  of  the  problem  required  the  help  of  the  computer 
support  staffs  at  NPRDC,  NPS,  and  the  DDN  Network  Information  Center.  This  was  costly  and 
time  consuming,  and  made  apparent  the  requirement  that  systems  such  as  the  ISN  have  for  support 
from  high-level  technical  personnel. 

TAC  hardware  failures.  We  are  aware  of  five  times  when  students  could  not  connect  to  a 
TAC,  but  this  problem  occurred  undoubtedly  many  times  without  being  brought  to  our  attention. 
TAC  failures  lasted  from  a  few  hours  to  several  days.  If  the  problem  was  reported  to  NPRDC  and 
did  not  correct  itself  in  a  day  or  so,  the  affected  students  were  given  the  telephone  number  of  a  dif¬ 
ferent  TAC  and  told  how  to  reconfigure  their  laptop  computer  to  use  the  new  telephone  number. 
Though  the  causes  for  all  the  TAC  failures  were  not  determined,  the  following  types  of  TAC  fail¬ 
ures  were  common: 

1.  All  telephone  lines  into  the  TAC  in  use,  resulting  in  a  busy  signal. 

2.  Using  a  new  TAC  telephone  number. 

3.  TAC  power  outage. 

4.  TAC  hardware  broken  or  undergoing  maintenance. 

5.  TAC  temporarily  or  permanently  disconnected. 

Telephone  line  failures.  The  only  recorded  telephone  system  problem  was  due  to  an  internal 
telephone  system  rather  than  the  external  telephone  company  lines.  This  problem  resulted  from 
incompatibility  between  the  ISN  software’s  requirements  for  touch-tone  dialing  capability  and  a 
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phone  system  using  newer  digital  dialing  with  special  signals.  The  student  was  unable  to  use  the 
ISN  in  this  phone  system  but  was  able  to  use  it  elsewhere. 

Student  Attrition 

Students  were  asked  during  the  post-course  debrief  interview  how  much  course  work  they  had 
completed.  Their  responses  were  used  to  place  them  into  one  of  two  categories: 

Category  A.  Student  had  taken  (or  attempted  to  take)  the  course’s  final  exam  within  eight 
weeks  of  enrollment.  An  attempt  to  take  the  final  exam  was  counted  as  success  because  the 
NPS  CE  program,  during  its  twilight,  did  not  always  keep  up  with  student  requests  for  tests  and, 
historically,  virtually  ail  students  who  took  exams  passed  them  with  a  grade  of  “B”  or  higher. 

Category  B.  Student  did  not  meet  the  criteria  of  Category  A. 

These  two  categories  provide  a  way  of  classifying  students  in  terms  of  attrition.  Category'  A 
students  were  classed  as  nonattrites  and  Category  B  students  as  attrites. 

Table  4  shows  the  number  of  students  in  each  category  by  group.  The  control  group  contains  a 
greater  proportion  of  attrites  than  the  ISN  group.  Table  4  suggests  that  the  ISN  reduced  attrition, 
but  this  conclusion  requires  statistical  verification.  The  small  number  of  subjects  precluded  the  use 
of  statistical  measures  based  on  simplifying  assumptions  about  underlying  population  distribu¬ 
tions.  Thus,  Fisher’s  exact  test  (Bradley,  1968)  was  used  on  the  data  in  Table  4  to  calculate  the  prob¬ 
ability  directly.  The  result  (the  probability  that  the  null  hypothesis  is  true)  is  not  statistically 
significant. 


Table  4 

Number  of  Nonattrites  and  Attrites  by 
Experimental  Group 


Exp. 

Group 

Nonattrites 

Attrites 

Totals 

ISN 

4 

6 

10 

Control 

5 

15 

20 

Totals 

9 

21 

30 

Student  mathematics  background  might  also  affect  attrition.  More  than  half  of  all  NPS  CE  cal¬ 
culus  students  had  previously  taken  calculus  and  enrolled  in  a  CE  course  for  review.  It  is  a  reason¬ 
able  hypothesis  that,  the  stronger  the  mathematics  background,  the  less  likely  the  student  would  be 
to  drop  out  of  the  class. 
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Table  5  shows  the  number  of  students  in  each  mathematics  background  category  by  group.  The 
“low”  group  had  three  or  less  semesters  of  calculus;  the  “high”  group  had  four  or  more  semesters 
of  calculus.  These  data  indicate  that  the  control  group  had  about  twice  as  many  students  with  high 
mathematics  background  as  did  the  ISN  group. 


Table  5 

Number  of  Students  in  Each  Mathematics 
Background  Category  by  Experimental  Group 


Exp. 

Group 

Math.  Background 

Low  High 

Totals 

ISN 

8 

2 

10 

Control 

13 

7 

20 

Totals 

21 

9 

30 

To  test  this  hypothesis,  a  one-way  between-groups  analysis  of  variance  was  calculated  to  de¬ 
termine  the  impact  of  mathematics  background  on  attrition.  The  result  (F(l,28)  =  2.35,  p  =  .14)  is 
not  statistically  significant,  but  suggests  (as  does  common  sense)  that  this  variable  plays  a  role  in 
attrition. 

Figure  5  shows  the  mathematics  background  of  nonaftrites  and  attrites  by  experimental  group. 
The  mathematics  background  of  nonattrites  is  slightly  higher  in  the  control  than  in  the  ISN  group. 
A  much  greater  difference  exists  for  the  attrites;  attrites  in  the  control  group  have  had  almost  twice 
as  much  mathematics  as  those  in  the  ISN  group.  This  leads  one  to  wonder  if  the  ISN  is  particularly 
beneficial  to  students  with  a  weak  mathematics  background.  Our  data  suggest  that  ISN  students 
with  weaker  mathematics  background  were  less  likely  to  attrite  than  were  controls. 

Table  6  shows  the  number  of  students  with  low  mathematics  background  in  terms  of  attrition 
category  and  group.  The  data  in  Table  6  enabled  us  to  test  the  hypothesis  that  the  ISN  is  particularly 
helpful  to  students  with  weaker  mathematics  backgrounds.  Fisher’s  exact  test  was  used  to  compute 
the  probability  that  the  null  hypothesis  is  true  for  the  data  shown  in  Table  6.  The  computed  proba¬ 
bility  is  not  statistically  significant. 
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Figure  5.  Mathematics  background  of  students  by  group  and  attrition  category. 


Table  6 

Number  of  Students  with  Low  Mathematics 
Background  by  Attrition  Category  and 
Experimental  Group 


Exp. 

Group 

Nonattrites 

Attrites 

Totals 

ISN 

3 

5 

8 

Control 

1 

12 

13 

Totals 

4 

17 

21 

Student  Attitudes 

During  the  student  debrief  interview,  students  were  asked  to  rate  various  dimensions  of  the 
class  they  had  taken  and  the  tutor  they  had  worked  with.  These  ratings  are  summarized  in  Tables 
7  (Course)  and  8  (Tutor)  for  each  dimension  rated  by  attrition  group. 
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Table  7  shows  student  ratings  of  class  utility,  overall  quality,  and  perceived  difficulty  on  a  rating 
scale  from  1  (low)  to  5  (high).  Overall,  the  ISN  group  tended  to  rate  the  class  more  favorably  than 
did  the  control  group.  Students  in  the  ISN  group  rated  class  usefulness  and  quality  higher,  and  class 
difficulty  lower,  than  did  controls.  None  of  the  between-groups  analyses  of  variance  for  these  com¬ 
parisons  was  statistically  significant. 


Table  7 

Student  Ratings  of  Class  Usefulness,  Quality,  and  Difficulty  by 
Experimental  Group  and  Attrition  Category 


ISN 

Control 

Class 

Dimension 

A 

B 

A&B 

A 

B 

A&B 

Usefulness 

4.0 

4.6 

4.3 

4.4 

3.6 

3.8 

Quality 

4.5 

3.6 

4.0 

3.6 

3.6 

3.6 

Difficulty 

3.5 

3.2 

3.3 

3.0 

3.7 

3.5 

Notes. 

1 .  Group  A  =  Nonattrites,  Group  B  =  Attrites. 

2.  Rating  scale  ranges  from  1  =  low  to  5  =  high. 


Table  8  shows  student  ratings  of  tutor  responsiveness,  knowledge  of  mathematics,  tutoring 
skill,  and  an  overall  rating  using  the  same  rating  scale  (from  1  =  low  to  5  =  high).  The  ISN  group 
tended  to  rate  all  dimensions  more  favorably  titan  did  the  control  group.  Between-groups  analyses 
of  variance  indicated  statistically  significant  differences  on  the  knowledge  of  mathematics  and  tu¬ 
toring  skill  dimensions. 

Though  the  differences  between  groups  on  most  of  these  ratings  are  quite  small,  they  tend  to 
indicate  that  students  in  the  ISN  group  were  more  favorably  disposed  toward  their  learning  expe¬ 
rience  than  were  those  in  the  control  group. 

The  most  important  finding  of  the  evaluation  is  that  it  is  possible  to  design  and  develop  a  system 
that  is  economical,  rugged,  and  easy  to  assemble;  has  simple  and  foolproof  documentation;  and  can 
be  operated  by  users  with  limited  computer  skills  and  technical  support.  Most  systems  that  do  what 
the  ISN  can  do  require  users  with  considerable  computer  sophistication  and  are  daunting  for  others. 
A  system  like  the  ISN-with  its  simple  user  interface,  high  degree  of  automation,  and  limited  set  of 
features-could  make  the  powerful  tool  of  electronic  networking  available  to  people  who  lack  com¬ 
puter  sophistication  or  who,  usually  for  technical  reasons,  choose  not  to  bother  mastering  more 
complex  messaging  systems. 
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Table  8 


Student  Ratings  of  Tutor  Responsiveness,  Mathematics  Knowledge,  Tutoring 
Skill,  and  Overall  Rating  by  Experimental  Group  and  Attrition  Category 


Class 

Dimension 

A 

ISN 

B 

A&B 

A 

Control 

B  A&B 

A&B 

Diff. 

Responsiveness 

5.0 

4.2 

4.6 

4.5 

4.0 

4.1 

0.5 

Math  Knowledge 

4.8 

5.0 

4.9 

5.0 

4.1 

4.2 

0.7* 

Tutoring  Skill 

4.8 

4.3 

4.5 

4.0 

3.5 

3.5 

1.0* 

Overall 

4.3 

4.3 

4.3 

4.0 

3.8 

3.9 

0.4 

Notes. 

1 .  Group  A  =  Nonattrites,  Group  B  =  Attrites. 

2.  Rating  scale  ranges  from  1  =  low  to  5  =  high. 

*p  < .05. 


It  came  as  no  surprise  that  the  ISN  required  technical  support,  although  the  actual  amount  was 
low.  The  tutor  was  the  most  time-intensive  user  of  the  ISN,  and,  if  the  number  of  students  were 
increased,  the  tutor’s  workload  would  increase.  However,  if  the  ISN  were  simply  used  for  commu¬ 
nication  between  individuals,  it  could  operate  with  relatively  little  technical  support. 

It  would  have  been  informative  if  the  attrition  and  attitude  data  had  indicated  the  utility  of  the 
ISN  for  instructional  purposes,  for  this  would  have  demonstrated  the  ISN ’s  usefulness  in  a  practical 
context;  i.e.,  with  college-educated  users  who  were  working  independently  on  a  fairly  difficult  self- 
paced  course  with  a  dismal  historical  completion  rate. However,  the  inconclusiveness  of  the  data 
does  not  diminish  the  success  of  users  in  actually  using  the  system. 

Hypothetical  ISN  Upgrade 

The  ISN  track  of  the  Communication  Networks  in  Training  project  was  terminated  after  the 
ISN  field  test.  During  the  early  part  of  the  field  test,  we  became  aware  of  several  limitations  of  the 
ISN  and  began  sketching  the  changes  to  be  made  to  a  second-generation  ISN.  This  work  was  car¬ 
ried  as  far  as  developing  a  working  prototype  of  the  ISN-II’s  user  interface  and  a  statement  of  work 
for  a  contractor  to  actually  develop  the  system.  The  system  was  not  developed  but  its  design  would 
overcome  some  shortcomings  of  the  prototype  ISN. 

The  upgrade  specifications  call  for  improvements  in  the  user  interface,  file  handling  capabili¬ 
ties,  text  editor,  and  message  handling  features,  but  its  two  most  significant  changes  are  the  use  of 
a  local  UNIX-based  host  computer  and  the  addition  of  teleconferencing  capabilities.  Many  of  the 
ISN’s  problems  resulted  from  the  use  of  a  host  computer  over  which  we  had  limited  control  and 
whose  operating  system  and  mail  handler  were  crude  compared  to  those  available  in  the  UNIX  en¬ 
vironment.  Use  of  a  local  UNIX  host  would  give  a  greater  degree  of  control  and  improved  message 
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handling.  The  prototype  ISN  permitted  one-on-one  communication  between  a  single  student  and 
a  single  tutor.  ISN-II  would  enable  any  user  to  address  any  message  to  any  single  user  or  combina¬ 
tion  of  users;  this  capability  would  enable  users  (students  or  other  types  of  users)  to  confer  with 
their  peers.  This  feature  would  also  make  the  ISN  useful  for  a  great  deal  more  than  tutoring;  e.g., 
for  teleconferencing  in  a  Navy  conferencing  network. 

Navy  Conferencing  Network 

The  Navy  does  not  have  a  conferencing  network  intended  for  general  use.  Since  various  Navy 
organizations,  groups,  and  entities  use  the  DDN  to  communicate,  it  can  be  argued  that  there  is  no 
need  to  change  the  status  quo.  Certainly  no  change  is  needed  if  the  intended  users  can  make  effec¬ 
tive  use  of  existing  communication  resources.  However,  substantial  evidence  suggests  that  current 
electronic  mail  systems  are  not  used  or  are  misused  because  of  their  complexity.  Simple  messaging 
systems  such  as  the  ISN  enable  users  to  do  perhaps  90  percent  of  what  typical  e-mail  users  need  to 
do  with  perhaps  10  percent  of  the  effort.  Existing  computer  conferencing  systems  (e.g.,  the  Army 
Forum  Network)  generally  have  primitive  user  interfaces  and  are  difficult  for  computer  novices 
and  occasional  users  to  operate.  Because  of  such  problems,  our  ISN  work  focused  on  very  simple 
and  easy-to-use  terminal  software  whose  design  might  serve  as  a  model  in  the  present  application. 
The  basic  design  problem  was  to  make  the  software  so  simple  and  user-friendly  that  it  could  work 
with  minimum  outside  support.  It  seems  to  have  succeeded  in  meeting  its  usability  objective.  As 
such,  it  would  be  a  good  model  for  a  Navy  Conferencing  Network  aimed  at  a  wider  audience  than 
is  currently  using  the  DDN  with  existing  software. 

CONCLUSIONS 

The  use  of  electronic  mail  appears  to  have  limited  application  for  instructional  delivery  in  the 
Navy.  However,  an  electronic  mail  system  with  the  simplicity  and  user  friendliness  of  the  ISN 
would  be  a  major  improvement  over  existing  electronic  mail  systems  and  would  be  useful  for  shar¬ 
ing  information  among  training  professionals  and  other  Navy  personnel. 

RECOMMENDATIONS 

1 .  Electronic  mail  should  not  be  considered  as  a  primary  instructional  delivery  medium  by  the 
Navy. 

2.  The  Chief  of  Naval  Education  and  Training  should  give  serious  consideration  to 
developing  a  Navy-wide,  DDN-based,  electronic-mail  network  for  sharing  information 
(“knowledge  networking”)  among  training  professionals  and  other  Navy  personnel.  The 
design  of  this  network  could  be  modeled  on  the  ISN. 
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APPENDIX 

THE  DEFENSE  DATA  NETWORK 
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THE  DEFENSE  DATA  NETWORK 


The  Defense  Data  Network  (DDN)  is  a  large  military  data  communications  network  operated 
for  the  Department  of  Defense  by  the  DDN  Program  Management  Office  (DDN  PMO)  of  the  De¬ 
fense  Communications  Agency  (DCA).  The  DDN  was  developed  in  1969  by  the  Defense  Ad¬ 
vanced  Research  Projects  Agency  (DARPA)  as  a  research  network,  and  later  was  used  for 
operational  military  purposes. 

The  DDN  links  military  and  research  computers  throughout  the  United  States  and  overseas. 
The  DDN  was  designed  to  connect  computers  produced  by  different  manufacturers  and  running 
different  operating  systems  to  the  network.  This  is  done  by  requiring  that  all  computers  connected 
to  the  network  use  a  common  communications  protocol,  called  Transmission  Control  Protocol/ 
Internet  Protocol  (TCP/IP).  All  computers  connected  to  the  DDN  are  required  to  provide  several 
network  services.  These  include  the  ability  to  log  onto  other  computers  in  the  network  (TELNET 
services),  the  ability  to  transfer  files  between  computers  (FTP  services),  and  electronic  mail 
services. 

The  DDN  includes  two  major  networks:  ARPANET  and  MILNET.  Both  are  unclassified 
military  networks,  but  they  have  different  uses.  The  ARPANET  supports  military  research  and 
development,  including  research  on  the  ARPANET  itself,  and  the  MILNET  is  an  operational 
network  that  supports  normal  day-to-day  military  data  communications. 

Users  of  the  DDN  have  two  ways  to  access  the  network.  They  can  access  the  network  through 
a  host  computer  connected  to  the  DDN.  In  general,  all  users  who  have  password  access  on  a  DDN 
host  computer  also  have  automatic  access  to  the  DDN.  Users  also  can  access  the  DDN  through  a 
Terminal  Access  Controller  (TAC).  A  TAC  is  a  general-purpose  connection  to  the  DDN  that  can 
be  used  with  a  variety  of  terminals  and  computers.  Scores  of  TACs  are  located  throughout  the 
United  States.  To  use  a  TAC,  users  must  have  either  a  terminal  connected  directly  to  a  TAC  or  a 
terminal  and  a  modem  through  which  they  can  dial  a  TAC.  By  entering  a  specific  logon  sequence, 
they  can  gain  access  to  the  TAC  and,  through  the  TAC,  access  computers  connected  to  the  DDN. 
Use  of  TACs  is  restricted  to  persons  with  a  valid  TAC  Access  Card.  Temporary  and  permanent 
TAC  Access  Cards  are  obtained  from  the  host  administrator  for  a  user’s  host  computer. 

DDN  is  logically  part  of  the  INTERNET,  a  collection  of  unaffiliated  networks  such  as  the 
NSFNET  and  the  Supercomputer  Network  CSNET.  These  various  networks  are  officially  or 
unofficially  gatewayed  together  at  well-connected  host  computers  around  the  United  States.  Today, 
more  than  100,000  host  computers  are  connected  to  the  INTERNET. 

Users  of  the  DDN  and  TACs  must  be  engaged  in  U.S.  government  business  or  applicable 
research  or  directly  involved  in  providing  operations  or  systems  support  for  government-owned  or 
e  wen.mcnt-sponsored  computer  communications  equipment.  The  network  is  not  available  for  use 
b  /  the  general  public. 

The  DDN  PMO  has  responsibility  for  overall  management,  operations,  and  policy  guidelines 
for  both  the  ARPANET  and  the  MILNET.  The  DDN  PMO  provides  many  services,  including 
keeping  the  network  running,  providing  user  assistance,  anticipating  growth  and  expansion,  and 
maintaining  network  security.  The  DDN  PMO  has  authorized  the  DDN  Network  Information 
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Center  (NIC)  to  provide  network  services  for  the  DDN,  and  the  NIC  is  usually  the  first  point  of 
contact  for  users  experiencing  problems  with  the  DDN.  The  NIC  is  located  at  SRI  International  in 
Menlo  Park,  California.  The  NIC  maintains  a  host  computer  system  containing  documents 
and  information  about  the  DDN. 
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